-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merge master-2.x commits 2023/07/01~2023/11/08 into main #18397
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…3.x [DOCFIX] Fix doc errors for Tencent Cloud EMR Please outline the changes and how this PR fixes the issue. Tencent Cloud EMR has updated some version information and needs to be synchronized to community documents Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#17747 change-id: cid-73d26a59de4f1023081ed0ffaf91cffcc2f8db0e
### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#17785 change-id: cid-657e47ef7657714a572a1f9811bbcc524bb8cfcc
### What changes are proposed in this pull request? Fix a deadlock. ### Why are the changes needed? On the Alluxio client, there is a monitor lock of `FileSystemContext` to sync individual client-to-master operations. One application thread can successfully hold the monitor lock (`alluxio.client.file.FileSystemContext`) in order to get worker list, but blocked by another “lock” `FileSystemContextReinitializer` (waiting for latch on on-going RPCs down to zero) for acquiring the block master client to really connect to master process (waiting for other callers involving `FileSystemContextReinitializer` to finish). On the other hand, another heartbeat thread "config-hash-master-heartbeat-0" is awaking periodically to sync with the master process to fetch the latest configuration. This thread detected the conf update and thus entered `FileSystemContextReinitializer` (bumping latch) but was blocked by waiting for the monitor lock of `alluxio.client.file.FileSystemContext` in order to get the master address. This PR moves `getMasterAddress` outside `reinit` block to avoid holding the `Reinitializer` object and wait for the monitor object of `FileSystemContext`. ### Does this PR introduce any user facing changes? No pr-link: Alluxio#18109 change-id: cid-0fab540f7dbef3661acb44e7472425add163b019
…and resolve a trivial conflict Fix local cache identifier If not including `modification time`, it may have consistency issues. NO pr-link: Alluxio#17514 change-id: cid-c9c2e91e53d4a95d4f3a165918fc23c02f623891
### What changes are proposed in this pull request? Delegate `getFileBlockLocation` to external file system in `LocalCacheFileSystem`. ### Why are the changes needed? Otherwise, `LocalCacheFileSystem` inherits the default behavior of `org.apache.hadoop.fs.FileSystem` which returns `localhost` only. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio#17672 change-id: cid-eb545dbd8ed42001d074fecfb9c8d6b118a559c1
### What changes are proposed in this pull request? Support read bytebuffer for non ByteBufferReadable input stream ### Why are the changes needed? Without this changes, a non ByteBufferReadable input stream can throw exception when the bytebuffer apis are called. ### Does this PR introduce any user facing changes? No pr-link: Alluxio#17982 change-id: cid-1363dbbeca327cbb7b4d39cea1afbb8c51905483
### What changes are proposed in this pull request? The v1 and v2 versions of proxy use uniform judgment conditions. pr-link: Alluxio#18371 change-id: cid-1842a0b96b5233f9a456e1aee7dbd95e2bd2d5e0
jiacheliu3
changed the title
Merge
Merge master-2.x commits 2023/07/01~2023/11/08 into main
Nov 8, 2023
master-2.x
from 2023/07/01 to 2023/11/08### What changes are proposed in this pull request? Support creating directories and files with default permissions based on configuration propertykey in HDFS API. ### Why are the changes needed? Alluxio Hdfs api hasn't the corresponding implementation of Mkdir without permission parameter. If it does not carry permission, the umask property in the configuration item shall prevail. ### Does this PR introduce any user facing changes? pr-link: Alluxio#18253 change-id: cid-6ddd2243bac00ebbdbdff1e731036c0d3d6228c8
… existed objects in OSS ufs Using the user of the OSS bucket to represent the user of the object loaded from OSS. Add the mapping way from OSS username to the custom username. Add the default permission mode for existing files loaded from OSS. the object loaded from OSS ufs will be null user, null group, and 777 by default. pr-link: Alluxio#18262 change-id: cid-7e56861c9def695876cde32d1e0aa453b512470d
Jackson-Wang-7
approved these changes
Nov 8, 2023
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great work, LGTM for my part, Thanks for this!
Xenorith
approved these changes
Nov 8, 2023
dbw9580
approved these changes
Nov 8, 2023
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
alluxio-bot, sync-merge this please |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Merge missing commits from master-2.x to main. The commits in 2023/07/01~2023/11/08 from main...master-2.x will be included by this PR.
We do this merge to catch missing fixes from
master-2.x
and catch the train beforemain
cuts a release.#17747 is not cherry picked because tencent cloud EMR doc is removed
#17755 is not cherry picked because DistLoadCliRunner has been removed in 3.x
#17758 is not cherry picked because MonoBlockStore has been removed in 3.x
#17641 is not cherry picked because the PR has already been in main
#17781 is not cherry picked because the PR has already been in main
#17722 is not cherry picked because the alluxio-fuse command has been changed a lot
#17489 is not cherry picked because audit log on master is no longer in 3.x
#17865 is not cherry picked because replication on job service is no longer in 3.x
#17858 is not cherry picked because it is already in main
#18090 is not cherry picked because generate-tarball has been rewritten in 3.x
#18091 is not cherry picked because the change is already in main
#17474 is not cherry picked because reconfiguration feature is not defined in 3.x
#17735 is not cherry picked because MonoBlockStore is no longer in 3.x
#18133 is not cherry picked because the issue is about master metadata and no longer relevant in 3.x
#17910 is not cherry picked because I prefer to do that manually
#17983 is not cherry picked because the web UI has been reworked
#17984 is not cherry picked because Mount/Unmount commands have been reworked in 3.x
#18103 is not cherry picked because worker cache metrics have been reworked in 3.x
#18185 is not cherry picked because the report command has been reworked in 3.x
#18222 is not cherry picked because Mount/Unmount operations have been reworked in 3.x
#18143 is not cherry picked because the change is already in main
#18303 is not cherry picked because the change is already in main
#18208 is not cherry picked because cache metrics have been reworked in 3.x
#17002 is not cherry picked because the owner has been notified separately
#18334 is not cherry picked because the bash scripts have been reworked in 3.x
#18326 is not cherry picked because the owner has been notified separately